{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chunked file arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chunked file arrays are chunked arrays whose chunks are stored in a file system. A store manager holds one or more chunks in memory, and flushes them to the file system when necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#include \"xtensor-io/xchunk_store_manager.hpp\"\n",
    "#include \"xtensor-io/xio_binary.hpp\"\n",
    "#include \"xtensor-io/xio_disk_handler.hpp\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "namespace fs = ghc::filesystem;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std::vector<size_t> shape = {4, 4};\n",
    "std::vector<size_t> chunk_shape = {2, 2};\n",
    "std::string chunk_dir = \"chunks\";\n",
    "double init_value = 5.5;\n",
    "\n",
    "fs::create_directory(chunk_dir);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A maximum of 2 chunks will be hold in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std::size_t pool_size = 2;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "auto a = xt::chunked_file_array<double, xt::xio_disk_handler<xt::xio_binary_config>>( \\\n",
    "    shape, \\\n",
    "    chunk_shape, \\\n",
    "    chunk_dir, \\\n",
    "    init_value, \\\n",
    "    pool_size);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This assigns to chunk `(1, 0)` in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a(2, 1) = 1.2;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This assigns to chunk `(0, 1)` in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a(1, 2) = 3.4;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the pool is full, this saves chunk `(1, 0)` to disk and assigns to chunk `(0, 0)` in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a(0, 0) = 5.6;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When `a` is destroyed or flushed, all the modified chunks are saved to disk. Here, only chunks `(0, 1)` and `(0, 0)` are saved, since chunk `(1, 0)` has not changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.chunks().flush();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "C++14",
   "language": "C++14",
   "name": "xcpp14"
  },
  "language_info": {
   "codemirror_mode": "text/x-c++src",
   "file_extension": ".cpp",
   "mimetype": "text/x-c++src",
   "name": "c++",
   "version": "14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
